Works great! Here is another demo on M1 Pro with F16 7B:

@FSSRepo Can you allow pushes to your branch so I can push some fixes:

$ ▶ git push FSSRepo HEAD:fixes 
error: Authentication error: Authentication required: You must have push access to verify locks
error: failed to push some refs to 'https://github.com/FSSRepo/llama.cpp'

@FSSRepo Can you allow pushes to your branch so I can push some fixes:

I don't have some restriction in that branch.

@ggerganov can you test full offloading model in your mac please? for confirm if there is some bug in faster generation scenario.

First off - this is awesome! Thank you @FSSRepo!

I am going to take the shameless opportunity here to request that adding support for speculative execution be considered here - that would make the first and only OSS LLM server I have come across that supports it out of the box!

Can this support the same generation parameters that are used in the existing example server?

Here is a 30B LLaMA Q4_0 serving 4 clients on M2 Ultra:

(I tried 7B and 13B, but the generation is so fast that I am not able to start the requests in parallel)

(I tried 7B and 13B, but the generation is so fast that I am not able to start the requests in parallel)

I want to add the option to cancel the streaming but when I use AbortControl in the frontend it causes error, I'm thinking add a get endpoint stop?slot=id to request release the sequence.

And one more example using the original server UI with 70B Q8_0 model:

And one more example using the original server UI with 70B Q8_0

Are you working in implement that ui or just reuse the endpoints?

Are you working in implement that ui

No, I just noticed it works and gave it a try. I thought you implemented it.

I want to add the option to cancel the streaming but when I use AbortControl in the frontend it causes error, I'm thinking add a get endpoint stop?slot=id to request release the sequence.

What kind of error?

How is this working? Are the instances sharing the same weights or the model needs to be loaded N times?

How is this working? Are the instances sharing the same weights or the model needs to be loaded N times?

Just divides the kv cache (context size) to a number of sequences, the limit is the context size because it is shared between clients.

This not reload the model, just once.

No, I just noticed it works and gave it a try. I thought you implemented it.

Only had reused the completion.js function. 😂

What kind of error?

 controller = new AbortController();
    const response = await fetch("http://localhost:8080/completion", {
      method: "POST",
      body: JSON.stringify(options),
      headers: {
        Connection: "keep-alive",
        "Content-Type": "application/json",
        Accept: "text/event-stream",
      },
      signal: controller.signal,
    });

function cancel() {
  if(controller) {
/* Anyway, even though I aborted it, the slot doesn't release, it continues to generate because the stream doesn't receive the signal that the connection was closed. 
Easy fix: create a stop endpoint to notify to slot release
*/
    controller.abort(); // when i call this function i get DOMException
  }
}

when I use AbortControl in the frontend it causes error,

"Note: When abort() is called, the fetch() promise rejects with a DOMException named AbortError." — https://developer.mozilla.org/en-US/docs/Web/API/AbortController

So that's probably normal. You can possibly try to catch the exception, like in the example there. I wouldn't think that would cause the connection not to abort properly though, so it not stopping generation on the server side is probably a different problem.

so it not stopping generation on the server side is probably a different problem.

I will fix it adding a GET endpoint stop_completion to notify the server release the slot.

@ggerganov can you merge the latest changes of master in this branch, please. I'm afraid to make a mistake again.

This example is very nice to have, but I'm thinking if we should try to directly implement the functionality in the existing server example. I'm not sure if I'll have time to do it myself though and I don't want to delay this super long. Also, the current implementation keeps one CPU core at 100% all the time which is not desirable.

I'll give this PR some more time to see if people would be interested in improving this. If not, we will probably merge this and add an item on the roadmap for the future.

This example is very nice to have, but I'm thinking if we should try to directly implement the functionality in the existing server example. I'm not sure if I'll have time to do it myself though and I don't want to delay this super long. Also, the current implementation keeps one CPU core at 100% all the time which is not desirable.

I'll give this PR some more time to see if people would be interested in improving this. If not, we will probably merge this and add an item on the roadmap for the future.

I will do it, i just want that you push the latest changes of master in my branch fixes, I don't know how to do that. Or if you teach me, I will be glad.

Or if you teach me, I will be glad.

You can add this repo as a remote in your fork (git remote add upstream https://github.com/ggerganov/llama.cpp.git) and then pull from it with your 'fixes' branch checked out (git pull --no-rebase upstream master).

@cebtenzzre Thank you so much!

@ggerganov "Do you want me to close this PR? I initially proposed this as a simple example to avoid the complexity of the server example.

@FSSRepo Let's see if we can make the other PR work and if so, we will close this one

Superseded #3677